Among the many forms of computer virtualization, machine or system virtualization has become common due to many known advantages. System virtualization involves abstracting the hardware resources of a computer and presenting the computer as virtual machines. A layer of software referred to as a hypervisor or virtual machine monitor (VMM) runs directly on the hardware of a computer. The hypervisor manages access to the hardware of the computer by virtual machines (VMs), which are also known as partitions, domains, or guests. Each VM is a software environment or construct capable of hosting its own guest operating system. The hypervisor manages sharing of the computer's hardware, in particular processing hardware and memory, by the VMs. The presence of a layer of software—the hypervisor—between the guest operating system and the computer hardware is mostly transparent to the guest operating system.
A developing technology for machine virtualization is the ability to directly assign hardware devices to VMs. This technology, referred to as direct device assignment (DDA), allows a virtualization layer (a term discussed in the Detailed Description) to hand over a hardware device to a VM and then the VM uses the hardware device directly without requiring substantial (if any) involvement of the virtualization layer to access and use the hardware device. That is, the hardware device is used by the VM without requiring virtualization abstraction or mapping by the virtualization layer, and, in some cases, neither the host nor other VMs use the hardware device, even when the virtualization layer has paused the VM or has not scheduled CPU time for the VM.
While DDA offers fast performance for the VM, there are disadvantages that only the instant inventor has appreciated. Most machine virtualization systems have functionality to control the execution state of VMs. Typical operations to control a VM's state include pausing, saving, restoring, migrating, and the like. Such operations are particularly useful in cloud environments. A cloud provider might need to alter the state of a VM transparently to the tenant or customer. For instance, a host machine might require a security update to the host/virtualization software. As only the inventor has appreciated, rather than informing a tenant or customer that a VM needs to be saved or shut down to allow a reboot of the host, the cloud provider would prefer to be able to suspend a tenant's VM transparently so that the host can be updated and rebooted without significantly interfering with operations of the tenant's VM and without requiring action by the tenant. Although it is theoretically possible for a virtualization system to preserve or pause the hardware state of a VM that is virtualized by the hypervisor (see, e.g., US patent publication 2014/0157264), it has not been possible for a virtualization system to perform such operations on hardware directly assigned to a VM and out of the purview of the hypervisor. That is, operations to control a VM's state have not been able to be performed on VMs with directly assigned hardware since the directly assigned device is owned by the VM.
There is a need to enable a machine virtualization system to control the state of a VM that takes into account the presence of hardware directly assigned to the VM.